Instrument-tissue Interaction Detection Framework for Surgical Video Understanding.

Instrument-tissue interaction detection task, which helps understand surgical activities, is vital for constructing computer-assisted surgery systems but with many challenges. Firstly, most models represent instrument-tissue interaction in a coarse-grained way which only focuses on classification and lacks the ability to automatically detect instruments and tissues. Secondly, existing works do not fully consider relations between intra-and inter-frame of instruments and tissues. In the paper, we propose to represent instrument-tissue interaction as ⟨instrument class, instrument bounding box, tissue class, tissue bounding box, action class⟩ quintuple and present an Instrument-Tissue Interaction Detection Network (ITIDNet) to detect the quintuple for surgery videos understanding. Specifically, we propose a Snippet Consecutive Feature (SCF) Layer to enhance features by modeling relationships of proposals in the current frame using global context information in the video snippet. We also propose a Spatial Corresponding Attention (SCA) Layer to incorporate features of proposals between adjacent frames through spatial encoding. To reason relationships between instruments and tissues, a Temporal Graph (TG) Layer is proposed with intra-frame connections to exploit relationships between instruments and tissues in the same frame and inter-frame connections to model the temporal information for the same instance. For evaluation, we build a cataract surgery video (PhacoQ) dataset and a cholecystectomy surgery video (CholecQ) dataset. Experimental results demonstrate the promising performance of our model, which outperforms other state-of-the-art models on both datasets.

Self-Healable and 4D Printable Hydrogel for Stretchable Electronics.

Materials with high stretchability and conductivity are used to fabricate stretchable electronics. Self-healing capability and four-dimensional (4D) printability are becoming increasingly important for these materials to facilitate their recovery from damage and endow them with stimuli-response properties. However, it remains challenging to design a single material that combines these four strengths. Here, a dually crosslinked hydrogel is developed by combining a covalently crosslinked acrylic acid (AAC) network and Fe3+ ions through dynamic and reversible ionically crosslinked coordination. The remarkable electrical sensitivity (a gauge factor of 3.93 under a strain of 1500%), superior stretchability (a fracture strain up to 1700%), self-healing ability (a healing efficiency of 88% and 97% for the mechanical and electrical properties, respectively), and 4D printability of the hydrogel are demonstrated by constructing a strain sensor, a two-dimensional touch panel, and shape-morphing structures with water-responsive behavior. The hydrogel demonstrates vast potential for applications in stretchable electronics.

Residual Physics and Post-Posed Shielding for Safe Deep Reinforcement Learning Method.

Deep reinforcement learning (DRL) has been researched for computer room air conditioning unit control problems in data centers (DCs). However, two main issues limit the deployment of DRL in actual systems. First, a large amount of data is needed. Next, as a mission-critical system, safe control needs to be guaranteed, and temperatures in DCs should be kept within a certain operating range. To mitigate these issues, this article proposes a novel control method RP-SDRL. First, Residual Physics, built using the first law of thermodynamics, is integrated with the DRL algorithm and a Prediction Model. Subsequently, a Correction Model adapted from gradient descent is combined with the Prediction Model as Post-Posed Shielding to enforce safe actions. The RP-SDRL method was validated using simulation. Noise is added to the states of the model to further test its performance under state uncertainty. Experimental results show that the combination of Residual Physics and DRL can significantly improve the initial policy, sample efficiency, and robustness. Residual Physics can also improve the sample efficiency and the accuracy of the prediction model. While DRL alone cannot avoid constraint violations, RP-SDRL can detect unsafe actions and significantly reduce violations. Compared to the baseline controller, about 13% of electricity usage can be saved.

MIcro-surgical anastomose workflow recognition challenge report.

BACKGROUND AND OBJECTIVE: Automatic surgical workflow recognition is an essential step in developing context-aware computer-assisted surgical systems. Video recordings of surgeries are becoming widely accessible, as the operational field view is captured during laparoscopic surgeries. Head and ceiling mounted cameras are also increasingly being used to record videos in open surgeries. This makes videos a common choice in surgical workflow recognition. Additional modalities, such as kinematic data captured during robot-assisted surgeries, could also improve workflow recognition. This paper presents the design and results of the MIcro-Surgical Anastomose Workflow recognition on training sessions (MISAW) challenge whose objective was to develop workflow recognition models based on kinematic data and/or videos. METHODS: The MISAW challenge provided a data set of 27 sequences of micro-surgical anastomosis on artificial blood vessels. This data set was composed of videos, kinematics, and workflow annotations. The latter described the sequences at three different granularity levels: phase, step, and activity. Four tasks were proposed to the participants: three of them were related to the recognition of surgical workflow at three different granularity levels, while the last one addressed the recognition of all granularity levels in the same model. We used the average application-dependent balanced accuracy (AD-Accuracy) as the evaluation metric. This takes unbalanced classes into account and it is more clinically relevant than a frame-by-frame score. RESULTS: Six teams participated in at least one task. All models employed deep learning models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), or a combination of both. The best models achieved accuracy above 95%, 80%, 60%, and 75% respectively for recognition of phases, steps, activities, and multi-granularity. The RNN-based models outperformed the CNN-based ones as well as the dedicated modality models compared to the multi-granularity except for activity recognition. CONCLUSION: For high levels of granularity, the best models had a recognition rate that may be sufficient for applications such as prediction of remaining surgical time. However, for activities, the recognition rate was still low for applications that can be employed clinically. The MISAW data set is publicly available at http://www.synapse.org/MISAW to encourage further research in surgical workflow recognition.

Robot-assisted flexible needle insertion using universal distributional deep reinforcement learning.

PURPOSE: Flexible needle insertion is an important minimally invasive surgery approach for biopsy and radio-frequency ablation. This approach can minimize intraoperative trauma and improve postoperative recovery. We propose a new path planning framework using multi-goal deep reinforcement learning to overcome the difficulties in uncertain needle-tissue interactions and enhance the robustness of robot-assisted insertion process. METHODS: This framework utilizes a new algorithm called universal distributional Q-learning (UDQL) to learn a stable steering policy and perform risk management by visualizing the learned Q-value distribution. To further improve the robustness, universal value function approximation is leveraged in the training process of UDQL to maximize generalization and connect to diagnosis by adapting fast re-planning and transfer learning. RESULTS: Computer simulation and phantom experimental results show our proposed framework can securely steer flexible needles with high insertion accuracy and robustness. The framework also improves robustness by providing distribution information to clinicians for diagnosis and decision making during surgery. CONCLUSIONS: Compared with previous methods, the proposed framework can perform multi-target needle insertion through single insertion point qunder continuous state space model with higher accuracy and robustness.

Shared control of a medical robot with haptic guidance.

PURPOSE: Tele-operation of robotic surgery reduces the radiation exposure during the interventional radiological operations. However, endoscope vision without force feedback on the surgical tool increases the difficulty for precise manipulation and the risk of tissue damage. The shared control of vision and force provides a novel approach of enhanced control with haptic guidance, which could lead to subtle dexterity and better maneuvrability during MIS surgery. METHODS: The paper provides an innovative shared control method for robotic minimally invasive surgery system, in which vision and haptic feedback are incorporated to provide guidance cues to the clinician during surgery. The incremental potential field (IPF) method is utilized to generate a guidance path based on the anatomy of tissue and surgical tool interaction. Haptic guidance is provided at the master end to assist the clinician during tele-operative surgical robotic task. RESULTS: The approach has been validated with path following and virtual tumor targeting experiments. The experiment results demonstrate that comparing with vision only guidance, the shared control with vision and haptics improved the accuracy and efficiency of surgical robotic manipulation, where the tool-position error distance and execution time are reduced. CONCLUSIONS: The validation experiment demonstrates that the shared control approach could help the surgical robot system provide stable assistance and precise performance to execute the designated surgical task. The methodology could also be implemented with other surgical robot with different surgical tools and applications.

Hand gesture guided robot-assisted surgery based on a direct augmented reality interface.

Radiofrequency (RF) ablation is a good alternative to hepatic resection for treatment of liver tumors. However, accurate needle insertion requires precise hand-eye coordination and is also affected by the difficulty of RF needle navigation. This paper proposes a cooperative surgical robot system, guided by hand gestures and supported by an augmented reality (AR)-based surgical field, for robot-assisted percutaneous treatment. It establishes a robot-assisted natural AR guidance mechanism that incorporates the advantages of the following three aspects: AR visual guidance information, surgeon's experiences and accuracy of robotic surgery. A projector-based AR environment is directly overlaid on a patient to display preoperative and intraoperative information, while a mobile surgical robot system implements specified RF needle insertion plans. Natural hand gestures are used as an intuitive and robust method to interact with both the AR system and surgical robot. The proposed system was evaluated on a mannequin model. Experimental results demonstrated that hand gesture guidance was able to effectively guide the surgical robot, and the robot-assisted implementation was found to improve the accuracy of needle insertion. This human-robot cooperative mechanism is a promising approach for precise transcutaneous ablation therapy.

New device for single-stage in-office secondary tracheoesophageal puncture: animal studies.

BACKGROUND: A new device is described for secondary tracheoesophageal puncture (TEP). We hypothesize that the device can be used to create the puncture, size, and insert the voice prosthesis in antegrade fashion in a single procedure suitable for office-based use. METHODS: A measurement and insertion device (MAID) was designed with the hypothesis in mind and tested on 3 pigs while they were under general anesthesia. RESULTS: The procedure was easy and successful in all 3 animals. The MAID allowed measurement of tracheoesophageal wall thickness and antegrade insertion of voice prosthesis with a single device. Transnasal esophagoscopy (TNE) allowed visualization of markings on the MAID within the esophagus to facilitate sizing. Intraesophageal visualization using TNE also enabled accurate positioning, reduced risk of posterior esophageal wall injury, and practically eliminated false passage formation. CONCLUSION: We believe the MAID can facilitate single-stage office-based secondary TEP in postlaryngectomy patients.

Development of a microclip for laryngeal microsurgery: initial animal studies.

OBJECTIVES/HYPOTHESIS: Mucosal closure following laryngeal microsurgery can be challenging due to limited access, and incisions are often left to heal without formal closure. This study explores the hypothesis that a bioabsorbable microclip can be developed as an alternative method to close mucosal incisions in laryngeal microsurgery. STUDY DESIGN: Animal study. METHODS: In vitro studies were performed to determine if a suitable clip could be designed using bioabsorbable materials. A porcine model was used to study behavior of the clip in vivo. RESULTS: We initially studied poly-ϵ-caprolactone but encountered difficulty creating a small clip with the necessary material strength. Using magnesium we were able to produce a clip sufficiently small to close vocal fold incisions. Magnesium is biocompatible, bioabsorbable, and malleable, and has been used to manufacture vascular stents and sutures. The magnesium microclip could be deployed rapidly using modified 2-mm micro-laryngeal cup forceps, which enable the clip to close in a circular shape. In vitro and in vivo tests showed the clip held securely to the vocal fold mucosa. Macroscopic and histologic studies showed no significant injury to the contralateral vocal fold. There was no evidence of lower airway injury after implanting clips into the lower airway. CONCLUSIONS: The bioabsorbable clip could be used to close selected incisions in laryngeal microsurgery. Currently we are working to further reduce the size of the clip and modify its bioabsorption properties to enable precise control of degradation.